How We Teach Are faculty predictions or item taxonomies useful for estimating the outcome of multiple-choice examinations?

نویسندگان

  • Jonathan D. Kibble
  • Teresa Johnson
چکیده

Kibble JD, Johnson T. Are faculty predictions or item taxonomies useful for estimating the outcome of multiple-choice examinations? Adv Physiol Educ 35: 396–401, 2011; doi:10.1152/advan.00062.2011.—The purpose of this study was to evaluate whether multiple-choice item difficulty could be predicted either by a subjective judgment by the question author or by applying a learning taxonomy to the items. Eight physiology faculty members teaching an upper-level undergraduate human physiology course consented to participate in the study. The faculty members annotated questions before exams with the descriptors “easy,” “moderate,” or “hard” and classified them according to whether they tested knowledge, comprehension, or application. Overall analysis showed a statistically significant, but relatively low, correlation between the intended item difficulty and actual student scores ( 0.19, P 0.01), indicating that, as intended item difficulty increased, the resulting student scores on items tended to decrease. Although this expected inverse relationship was detected, faculty members were correct only 48% of the time when estimating difficulty. There was also significant individual variation among faculty members in the ability to predict item difficulty ( 2 16.84, P 0.02). With regard to the cognitive level of items, no significant correlation was found between the item cognitive level and either actual student scores ( 0.09, P 0.14) or item discrimination ( 0.05, P 0.42). Despite the inability of faculty members to accurately predict item difficulty, the examinations were of high quality, as evidenced by reliability coefficients (Cronbach’s ) of 0.70–0.92, the rejection of only 4 of 300 items in the postexamination review, and a mean item discrimination (point biserial) of 0.37. In conclusion, the effort of assigning annotations describing intended difficulty and cognitive levels to multiple-choice items is of doubtful value in terms of controlling examination difficulty. However, we also report that the process of annotating questions may enhance examination validity and can reveal aspects of the hidden curriculum.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Are faculty predictions or item taxonomies useful for estimating the outcome of multiple-choice examinations?

The purpose of this study was to evaluate whether multiple-choice item difficulty could be predicted either by a subjective judgment by the question author or by applying a learning taxonomy to the items. Eight physiology faculty members teaching an upper-level undergraduate human physiology course consented to participate in the study. The faculty members annotated questions before exams with ...

متن کامل

Utility of Complex Alternatives in Multiple-Choice Items: The Case of All of the Above

This study investigated the utility of all of the above (AOTA) as a test option in multiple-choice items. It aimed at estimating item fit, item difficulty, item discrimination, and guess factor of such a choice. Five reading passages of the Key English Test (KET, 2010) were adapted. The test was reconstructed in 2 parallel forms: Test 1 did not include the abovementioned alternative, whereas Te...

متن کامل

Influence of Assessment Method Selection in Studying and Learning Approaches: Is It Necessary to Change Assessment Style?

Background: It is important to learn how to study for different examinations. The objective of the current study is to explore whether the assessment method selection would significantly affect the studying and learning approaches of students. Methods: This descriptive-analytical research consisted of 191 first-year undergraduate nursing students from three nursing schools and was conducted dur...

متن کامل

The Impact of Correction for Guessing Formula on MC and Yes/No Vocabulary Tests' Scores

A standard correction for random guessing (cfg) formula on multiple-choice and Yes/Noexaminations was examined retrospectively in the scores of the intermediate female EFL learners in an English language school. The correctionwas a weighting formula for points awarded for correct answers,incorrect answers, and unanswered questions so that the expectedvalue of the increase in test score due to g...

متن کامل

Partial Knowledge in Multiple-Choice Testing

The intent of this study was to discover the nature of (partial) knowledge as estimated by the multiple-choice (MC) test method. An MC test of vocabulary, including 20 items, was given to 10 participants. Each examinee was required to think aloud while focusing on each item before and while making a response. After each test taker was done with each item, s/he was ...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2011